The invention relates to human/computer interfaces to visual data and more particularly to systems that must display a larger amount of visual data than may be conveniently displayed in a single conventional computer monitor. The present invention uses virtual reality techniques to provide instantaneous and intuitive access to large fields of visual data, and to provide visually-impaired users with enhanced access to enlarged visual data.
Among the visually-impaired population, the most common approach to computer access is specialized software. and/or hardware that enlarges the image displayed on the computer monitor. This is because simpler solutions such as moving closer to the monitor, using a larger monitor, adding an optical screen magnifier, or using a spectacle-mounted telescopic system provide either limited magnification or a very limited viewing field. Examples of commercially-available screen enlargers include LP-DOS by Optelec (Westford, Mass.), Zoomtext by Ai Squared (Manchester Center, Vt.), MAGic by Microsystems Software (Framingham, Mass.), and Magnum by Arctic Technologies (Troy, Mich.). In addition, simplistic screen enlargement modules are included in both the Microsoft Windows and Apple Macintosh operating systems.
These conventional computer display magnification solutions operate by magnifying the original image of a software application""s output to a xe2x80x9cvirtual pagexe2x80x9d whose size is much larger than the physical monitor. For example, with a magnification of 10, a standard 8.5xe2x80x3xc3x9711xe2x80x3 page would be approximately 7 feet wide by 9 feet tall. The visually-impaired user then operates the computer by using a mouse, joystick, or cursor keys to control which portion of the virtual page is shown on the monitor at any given point in time. Since the monitor is fixed, the user is in essence moving the virtual page across the monitor, in a manner analogous to that used in closed-circuit television (CCTV) systems for magnifying book pages.
In most cases, conventional screen magnification is performed completely in software running on the host computer""s central processing unit (CPU). While this provides a very low-cost solution, the data to be shown on the display must be rendered in its entirety whenever the user pans to a new location within the enlarged image. This can result in lags between commanding the computer to pan and seeing the new image. To overcome this problem, the entire virtual image can be rendered and stored in a video display buffer. Then, as the user selects a portion of the image for viewing, the required portion of the data can be quickly read out of the display buffer and sent to the display device. An example of such a hardware-accelerated screen magnifier is the Vista by Telesensory, Inc. (Mountain View, Calif.). This technique is a form of hardware acceleration known as image deflection.
Unfortunately, there are two basic shortcomings to the conventional approach, even with hardware acceleration. The first problem is spatial orientation, in that it is difficult to determine where on the page one""s view is directed at any given time. This occurs because the monitor does not move, and there are no other visual cues to indicate where on the virtual page one""s line of sight is facing. This spatial orientation problem is exacerbated for high magnifications and for portable systems employing small display monitors. For example, one study (Goodrich, et. al.) found mean magnifications of 15.48xc3x97for nearly 100 experienced users of closed-circuit television devices. At 15xc3x97, a 15xe2x80x3 monitor can only display about 1% of a standard 8.5xe2x80x3xc3x9711xe2x80x3 page, making most computer work essentially impossible for such users. The problem is further exacerbated by the emergence of graphically-intensive computing regimes such as Microsoft Windows and the Internet World Wide Web, where individual graphic elements may be magnified to become larger than an instantaneous viewing window, or may be automatically generated outside of the user""s instantaneous viewing window without the user""s awareness.
The second fundamental problem in the conventional approach is dynamic control, in that all of the various control schemes for navigating about the page are cumbersome, confusing, and slow. This is because the navigation methods are unintuitive, relying on such logic as xe2x80x9cuse joystick to move cursor around screen, and when cursor reaches the edge of the screen, the next portion of document in that direction will be displayed.xe2x80x9d Alternatively, some screen enlargers maintain the cursor at the center of the screen, and require the user to position a desired insertion point over the cursor by moving the entire virtual page with a mouse or joystick. In all cases, dynamic control is not only unintuitive, but requires use of at least one hand, which negatively impacts productivity, and may make use by physically-impaired users difficult or impossible.
Together, these spatial orientation and dynamic control problems were termed the xe2x80x9cfield navigationxe2x80x9d problem in the National Advisory Eye Council""s 1994-98 National Plan (Legge, et. al.), in which the Low Vision and its Rehabilitation Panel identified this area as a particularly promising opportunity for new technologies.
One promising new technology that is now maturing is virtual reality, which is typically defined as a computer-generated three-dimensional environment providing the ability to navigate about the environment, turn one""s head to look around the environment, and interact with simulated objects in the environment using a control peripheral.
In a virtual reality system, the user is xe2x80x9cimmersedxe2x80x9d in a synthetic environment, in which virtual objects can be located anywhere in the user""s physical space. The user views these objects by wearing a head-mounted display (HMD), which uses an optical system to cause a tiny display source such as a cathode ray tube or liquid crystal display to appear as a large display screen several feet in front of the user. Since the display source (or sources in the case of two eyes) is fixed to the user""s head, the display is viewable regardless of where the user points his line-of-sight. The user also wears a head-tracker, which senses the direction the user is facing, and sends this information to the host computer. The computer uses this data to generate graphics corresponding to the user""s line of sight in the virtual environment. This approach to human/computer interfaces was first conceived by Ivan Sutherland in 1966 for use in military simulators, and was first commercialized in the form of the Eyephone head-mounted display by VPL Research in the late 1980s.
Prior art in this area includes a wide range of relevant patents describing low-vision aids, improved virtual reality systems and components such as HMDs and head-trackers, but none which embody or anticipate the present invention.
In the field of low-vision aids, U.S. Pat. No. 4,227,209 issued Oct. 10, 1980 discloses an electronic sensory aid for visually-impaired users including an image sensor and a display array, wherein the degree of magnification provided in the display array may be adjusted by changing the number of display elements corresponding to each sensor array element. For use in electronic sensory aid applications requiring a large depth of focus, an improved image capture approach is disclosed in U.S. Pat. No. 5,325,123 issued Jun. 28, 1994, in which the imaging camera includes an opaque stop with a small aperture, thus allowing the magnification to be adjusted by moving the camera towards or away from the object to be magnified. A non-electronic sensory aid is disclosed in U.S. Pat. No. 4,548,485 issued Oct. 22, 1985, in which an XY stage is used to move textual material across an optical viewing system that captures a portion of the textual material for enlargement.
In U.S. Pat. No. 5,125,046 issued Jun. 23, 1992, and U.S. Pat. No. 5,267,331 issued Nov. 30, 1993, an improved imaging enhancer for visually-impaired users is disclosed in which an image is captured, digitized, and electronically enhanced to increase contrast before displaying the imagery. An improvement to this approach using a head-mounted display is disclosed in U.S. Pat. No. 5,359,675, issued Oct. 25, 1994.
In the field of virtual reality systems, U.S. Pat. No. 5,367,614 issued Nov. 22, 1994 to Bisey discloses a three-dimensional computer image display system using an ultrasonic transceiver head-tracking system to control a three-dimensional display to cause the image to change its perspective in response to head movements. In addition, U.S. Pat. No. 5,442,734 issued Aug. 15, 1995 to Murakami discloses a virtual reality system incorporating a head-mounted display, head-tracker, and image processing system in which predictive tracking algorithms are used to differentially update portions of the display field to provide more rapid updating of those portions of the display field corresponding to the center of the user""s visual field. In U.S. patent application No. 07/621,446 (pending) filed by VPL Research, Inc., a virtual reality system is disclosed in which spatialized audio cues are generated to provide real-time feedback to users upon successful completion of manual tasks such as grasping virtual objects using a sensor-laden glove input device.
In the head-mounted display field, U.S. Pat. No. 5,003,300 issued Mar. 26, 1991 to Wells discloses a raster-based head-mounted display that may be used to display an image to either eye. U.S. Pat. No. 5,151,722 issued Sep. 29, 1992 to Massof discloses a video-based head-mounted display featuring a unique folding optic configuration so that the device may be worn like a pair of glasses. U.S. Pat. No. 5,281,957 issued Jan. 25, 1994 to Schoolman discloses a portable computer system incorporating a head-mounted display that may be hinge-mounted to an eyeglass frame so that the display may be folded up out of the way for viewing the physical environment. A wide variety of additional patents in the area of specific design improvements for head-mounted display devices exists, however, the specific head-mounted display design approach employed to effect the present invention is not critical, so long as image quality, brightness, contrast, and comfort are maintained at high levels.
In recent years, there have been several attempts made to apply head-mounted displays to the problems of enhancing imagery for visually-impaired users. One such effort has resulted in the Low-Vision Enhancement System LVES) developed by Johns Hopkins University and marketed commercially by Visionics (Minneapolis, Minn.). The LVES device incorporates a head-mounted display with integrated cameras and an image processing system. The cameras generate an image of whatever is positioned directly in front of the user, and the image processing system enlarges the image and performs enhancement functions such as contrast enhancement. While the LVES device can provide magnified imagery of real-world objects to some visually-impaired users, it suffers from several shortcomings compared to the present invention. First, the LVES does not incorporate a head-tracker to provide a hands-free means for navigating within computer data. Further, the LVES suffers from a jitter problem exactly analogous to that experienced by users of binoculars or telescopes. In simple terms, any jitter in the user""s line-of-sight is magnified by the same factor as the imagery, which causes the image provided to the user to appear unsteady.
U.S. Pat. No. 5,109,282 issued Apr. 28, 1992 to Peli discloses a novel image processing method for converting continuous grey tone images into high resolution halftone images, and describes an embodiment of the method applicable to presentation of enlarged imagery to visually-impaired users via a head-mounted display. In this device, the imagery is generated by a conventional camera manually scanned across printed text as is common in closed-circuit television systems for the visually-impaired. The head-mounted display is a Private Eye by Reflection Technologies (Waltham, Mass.), which employs a linear array of light-emitting diodes converted to the impression of a rectangular array by means of a scanning mirror. In the disclosed device, benefits of using a head-mounted display for low-vision access to printed material in portable situations are discussed, including the larger visual field, higher visual contrast, lighter weight, and smaller physical size provided by an HMD compared to a portable conventional television monitor. However, no connection to a computer for viewing computer-generated imagery is disclosed or anticipated, and no incorporation of a head-tracking device is disclosed or anticipated.
In the tracker art, a variety of tracking approaches and applications have been conceived and constructed. U.S. Pat. No. 5,373,857 issued Dec. 12, 1994 to Travers, discloses a head-tracking approach for the yaw degree of freedom in virtual reality applications consisting of a magnetic sensor disposed on a headset to produce a displacement signal relative to angular displacement of the head set with respect to the earth""s magnetic field. A more sophisticated approach has been developed by the Massachusetts Institute of Technology (MIT), in which an analogous magnetic sensor is used to correct drift in a much faster differential sensor such as an accelerometer, which sensors together provide extremely rapid response and high accuracy within a single package. The MIT approach, believed to be patent-pending, additionally incorporates differential sensors to detect changes in the pitch and roll degrees of freedom, which sensors may also be corrected using slower absolute sensors such as liquid-filled capacitive tilt sensors.
Also within the tracker art, a number of devices have been disclosed which sense head movement for purposes of controlling positioning of a cursor or mouse pointer within the viewable portion of a conventional display monitor. U.S. Pat. No. 4,209,255 issued Jun. 24, 1980 to Heynau discloses a system for pilots employing a light-emitting diode mounted on the pilot""s head, with photodiodes located on the display to sense the tapered energy field from the light-emitting diode for purposes of determining the pilot""s aimpoint within the display.
U.S. Pat. No. 4,565,999 issued Jan. 21, 1986 to King discloses a cursor control system for use with a data terminal wherein a radiation source and a radiation sensor are used to determine changes in a user""s head position for purposes of controlling cursor position on the screen.
U.S. Pat. No. 4,567,479 issued Jan. 28, 1986 to Boyd discloses a directional controller for video or computer input by physically-impaired users consisting of a series of mercury switches disposed in proximity to a user""s head, wherein movements of the user""s head are sensed and converted into cursor control commands. This device also employs a pressure switch activated by the user""s mouth which can provide a further control signal such as that generated by clicking a mouse button.
U.S. Pat. No. 4,682,159 issued Jul. 27, 1987 to Davison discloses an apparatus and method for controlling a cursor on a computer display that consists of a headset worn by the user, and a stationary ultrasonic transmitter for emitting sound waves which are picked up by receivers in the headset. These sound waves are compared for phase changes, which are converted into positional change data for controlling the cursor.
U.S. Pat. No. 5,367,315 issued Nov. 22, 1994 to Pan discloses an infrared-light based system that indicates head and eye position in real time, so as to enable a computer user to control cursor movement on a display by moving his or her eyes or head. The device is intended to emulate a standard mouse, thereby allowing use of the presently available software and hardware.
While the above examples demonstrate a well-developed art for controlling computer cursors via head movement, none disclose or anticipate application of head-controlled cursor movement within a head-mounted display, and none anticipate an approach such as the present invention wherein the cursor remains fixed at a particular position within the display while the displayed data is moved instead of the cursor. Movement of displayed data within a head-mounted display in response to head movement has heretofore been used only within virtual reality systems designed for simulating sensory immersion within three-dimensional computer simulations. In such applications, cursors or mouse pointers are not controlled by head movement, but are generated when required through the use of a separate hand-controlled input device.
While virtual reality is still a developmental technology involving exotic graphics hardware, specialized software, and long integration cycles, the concept of closing a control loop between head-tracker data and HMD imagery can be implemented analogously for viewing arbitrary computer data instead of specially-constructed virtual environments. For normally sighted individuals, this could be beneficial by providing a large virtual computer desktop surrounding the user, which can provide simultaneous access to a larger amount of visual data than is possible using the small virtual desktops currently provided on common computing platforms such as Macintosh and Windows. For visually-impaired individuals, head-tracked HMD display techniques can be used to conveniently access a magnified virtual page, and thus enable productive computer use by nearly 1,000,000 new users.
It is therefore an object of the current invention to solve the field navigation problem by combining virtual reality display techniques originally developed for military flight simulators with screen magnification techniques, in order to create a novel and intuitive display interface for visually impaired users.
It is another object of the current invention to provide an intuitive computer display interface allowing the user to automatically achieve proper spatial orientation by directly coupling the user""s head orientation to the displayed portion of a magnified virtual page.
It is a further object of the current invention to provide an intuitive portable computer display interface allowing the user to automatically control the position of a cursor or mouse pointer on a computer-generated virtual page by directly coupling the user""s head movements to movements of a cursor across the virtual page, thus freeing the user""s hands for other tasks.
It is an additional object of the present invention to provide hands-free instantaneous selection from between many concurrently active computer applications by changing one""s line-of-sight from one application window""s virtual location to another.
It is yet another object of the present invention to provide and maintain a cursor at a user-selectable position within the user""s field-of-view, in order to support use of the virtual computer display by users with arbitrary, non-central preferred retinal loci.
It is still another object of the present invention to alert the user to events occurring outside of the user""s instantaneous field-of-view through the use of spatialized audio alerts perceived to originate from the direction of the event, thus causing the user to turn and look in the direction of the event.
It is yet a further object of the present invention to provide effective operation at magnifications much greater than those possible using fixed monitors, by using a novel technique known as spatial compression.
It is still another object of the present invention to provide improved scrolling of imagery across the user""s field-of-view, through application of smoothing, thresholding, prediction, and drift compensation algorithms to improve response to raw data representing the user""s instantaneous line of sight.
It is still a further object of the present invention to provide a computer display for visually-impaired users that is convenient, lightweight, low-cost, minimally power hungry, and capable of portable operation without degraded performance.
It is another object of the present invention to provide a means for visually-impaired users to view enlarged video imagery in real time over an expanded field-of-regard, thus reducing jitter compared to head-mounted closed-circuit television systems.
In accordance with the present invention, there has been devised a xe2x80x9cvirtual computer monitorxe2x80x9d (VCM) which is broadly comprised of a head-mounted display means worn by the user, a head-orientation sensing means worn by the user, and software means for interfacing these devices to a host computer such that the user""s head orientation data is processed to determine which portion of an arbitrary software application""s output imagery to display. By properly matching the angle of head rotation to the extent of scrolling across the magnified image, the image can be made to appear fixed in space. The particular location of the portion of the virtual image which is actually being seen by the user is dependent upon the direction in which the user looks. As the user looks to the right, the portion of the virtual image being seen by the user is to the right of the portion of the virtual image previously being seen by the user. Similarly, as the user looks up, the portion of the virtual image being seen by the user is above the portion of the virtual image previously seen by the user. Upon initialization of the VCM device, the user triggers calibration between the user""s straight-ahead line of sight and the center of the virtual page. From then on, the user can rotate her head left, right, up, and down to visually scan across the page in corresponding directions. The overall impression is analogous to a normally sighted person scanning across a newspaper page.
As applied to a computer interface device for the visually-impaired, the VCM software provides a magnification adjustment to allow each user to achieve adequate visual resolution without needlessly reducing his instantaneous viewing field. The software also provides a cursor, which nominally remains positioned at the center of the HMD physical field regardless of head movements so that the cursor can be positioned anywhere upon the virtual page by turning to face that location. A further adjustment allows setting the fixed cursor location to any arbitrary position in the HMD device""s physical field, so that users with unusable portions of their visual fields can select an alternative preferred retinal loci instead of the center. A software selection also provides an overview display, which shows a reduced-magnification image of the entire virtual page, with a bold black box highlighting the outline of the instantaneous field within the entire field.
An additional important feature is the ability to temporarily adjust the cursor position in real-time using a controller peripheral such as a joystick or mouse. This feature allows fine positioning of the cursor within the field by temporarily locking the head-tracking system to freeze a portion of the virtual page on the physical display, while the controller is used to move the cursor in small increments.
An additional important feature is the ability to display image components in addition to the cursor at fixed points in the physical display, which allows menus or other icons to remain in the user""s instantaneous viewing field at all times while scrolling across image content.
An additional important feature resides in the ability to reduce the lag between a head motion and display of the new direction""s image by using image deflection, thresholding, smoothing, prediction, and a novel drift compensation technique to reduce display xe2x80x9cswimmingxe2x80x9d, which is caused whenever imperfect head orientation sensing causes the displayed image to not appear fixed in real-space.
An additional important feature resides in the ability to magnify images by extremely large factors using spatial field compression, where the displayed image is scrolled across the physical display at a faster rate than the head is turned. This enables use by individuals with limited head motion, and allows magnification to levels that would otherwise require turning completely around to view edges of the image.
An additional important feature resides in the use of a partially immersive HMD, which avoids simulation sickness by allowing the user to maintain a constant frame of reference in the physical world since real objects can be seen around one or more edges of the display.
It is therefore an advantage of the current invention that it solves the field navigation problem by combining virtual reality display techniques originally developed for military flight simulators with screen magnification techniques, in order to provide a novel and intuitive display interface for visually impaired users.
It is another advantage of the current invention that it provides an intuitive computer display interface allowing the user to automatically achieve proper spatial orientation by directly coupling the user""s head orientation to the displayed portion of a magnified virtual page.
It is a further advantage of the current invention that it provides an intuitive computer display interface allowing the user to automatically control the position of a cursor or mouse pointer on a computer-generated virtual page by directly coupling the user""s head movements to movements of a cursor across the virtual page, thus freeing the user""s hands for other tasks.
It is an additional advantage of the present invention that it provides hands-free instantaneous selection from between many concurrently active computer applications by changing one""s line-of-sight from one application window""s virtual location to another.
It is yet another advantage of the present invention that it provides and maintains a cursor at a user-selectable position within the user""s field-of-view, in order to support use of the virtual computer display by users with arbitrary, non-central preferred retinal loci.
It is still another advantage of the present invention that it alerts the user to events occurring outside of the user""s instantaneous field-of-view through the use of spatialized audio alerts perceived to originate from the direction of the event, thus causing the user to turn and look in the direction of the event.
It is yet a further advantage of the present invention that it provides effective operation at magnifications much greater than those possible using fixed monitors, by using a novel technique known as spatial compression.
It is still another advantage of the present invention that it provides improved scrolling of imagery across the user""s field-of-view, through application of smoothing, thresholding, prediction, and drift compensation algorithms to improve response to raw data representing the user""s instantaneous line of sight.
It is still a further advantage of the present invention that it provides a computer display for visually-impaired users that is convenient, lightweight, low-cost, minimally power hungry, and capable of portable operation without degraded performance.
It is another advantage of the present invention that it provides a means for visually-impaired users to view enlarged video imagery in real time over an expanded field-of-regard, thus reducing jitter compared to head-mounted closed-circuit television systems.